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1. INTRODUCTION 

Software is an important aspect that has an impact on human life, such as work, daily activity, 
finance, and other essential fields. There are some steps to build software, and developers must have a clear 
understanding of the reguirements of the software. However, the developer tends to build software without 
having a clear understanding of the reguirements in detail because for them having understood reguirements 
in detail is a waste of time, in which requirements are constantly changing [1]. These things lead to software 
failure because when the developer does not have a clear understanding of the system, an error can occur in 
system design and the next steps [2]. 

Requirements engineering is an activity that has the aim to know, understand, analyze, and 
documenting what are the requirements that needed by stakeholder [1], so involvement of stakeholders is 
crucial in software project especially in requirements engineering process [1, 3]. One of the steps 
requirements engineering is requirements elicitation which user story is one of its practice. User stories 
introduced the first time to extreme programing in agile development methods and started used in other 
methods, such as Scrum, [4]. User stories focus on interaction with the user, so users can participate and 
describe the functions that can be useful for stakeholders. A chatbot is a computer agent that can interact with 
the user and make human-bot conversation feels like they were a human-human conversation. Based on 
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previous research, natural language processing used for chatbot can interact and understand like a human 
[5-7]. Based on a survey by Oracle, more than 70% of the respondents said that people or businesspeople 
already used chatbot or have a plan to use a chatbot in 2020 [8]. When using a chatbot, the customer does not 
need to waste their time to meet face to face to get specific service which makes users save the cost, these 
things make chatbot has brought positive impact for both customers and the company [9]. 

Sending messages is the core of the mobile experience [10], and the increase in internet and mobile 
devices in the present makes the interaction between humans easier [11]. Based on a survey in 2017, mobile 
device users in Indonesia already reach 341,4 million [12]. Artificial mark-up language (AIML) 1s 
an XML-based mark-up language that used for brain or knowledge of the chatbot which used the first time in 
ALICE bot [13, 14]. In other researches, using AIML for the brain of the chatbot is a success in creating 
interaction between bot and user. 

The Nazief & Adriani algorithm is a stemming algorithm commonly used for Bahasa Indonesia, 
which is considered as the qualifief one, compared to other stemming algorithms [15]. That algorithm used 
because chatbot using Bahasa Indonesia and Nazief & Adriani algorithm have higher accuracy for 
the Indonesian language stemmer than other algorithms such as porter [16]. Other works have also shown 
the potential of Nazief & Adriani algorithm in stemming natural languages in Bahasa Indonesia [17—19]. 
Furthermore, chatbot for related works of literature concerning the development chatbot in Bahasa Indonesia 
has also shown the use of chatbot in various fields [20—22]; however, the use of chatbot and Nazief & 
Adriani algorithm in the requirements engineering field 1s still rare to be found. 

This research aims to bridge the gap between the stakeholders and the software engineers in terms of 
requirements gathering by developing a chatbot for the stakeholders to chat with when submitting 
preliminary requirements and/or user feedbacks regarding a software product. The chatbot uses AIML as its 
knowledge base, Nazief & Adriani stemming algorithm, and follows the User Stories format which will 
require the users to provide information regarding his/her role, needs, and business value in a natural 
conversation. The bot’s performance was measured using several chatbot metrics, and a preliminary user 
acceptance test following the technology acceptance model was conducted for evaluation. 


2. RESEARCH METHOD 

In this research, the chatbot application built using the Nazief & Adriani stemming algorithm for 
the stemming process and AIML is used for knowledge of the chatbot. Chatbot used for gathering software 
requirements using user stories. 


2.1. Chatbot design 

In building the Android application with the chatbot included, the object-oriented programming 
approach is used. UML diagrammatic notations are used to design the application. Figure 1 shows the use 
case diagram of the chatbot. There are two kinds of users, the generic user, and the software developer. 
The generic users could perform two main activities which are to view the list of chat rooms in his/her 
application, and also to give user stories by communicating with the chatbot. The developers could do 
anything the generic user can do, and also, he/she could also view the submitted user stories as 
the requirements gathered for the corresponding application. 


2.2. AIML 

Our research uses AIML (Artificial Intelligence Markup Language) used for the knowledge (brain) 
of the chatbot. In AIML, for one tag category usually, it consists of tag pattern and tag template. Tag pattern 
is used for matching user input and tag template used for output from the bot. Figure 2 shows an example of 
the AIML of the chatbot application. That example shows the knowledge of the chatbot for users to input 
their role. After the user inputs their role, the submitted data will be caught by the wildcard symbol in tag 
pattern and will be set in variable temp. After that, in order to get a response from the chatbot, the value of 
the variable temp will be checked. 


2.3. Text pre-processing 

Text preprocessing process is done before parsing the message to the brain file. Firstly, 
normalization is done, which message will be converted to lower case. After normalizing the message, 
tokenizing will be done. Tokenizing is a process that split the message into tokens. Each token will be 
checked if there is a mistyped word. Process stop word removal will be done after checking the typo word 
after that stemming will be done with Nazief & Adriani stemming algorithm. Stemming starts by checking 
the combination prefix-suffix process. If a common combination detected, the suffix is then checked and 
stemmed, and after that, the prefix will be checked and stemmed, and finally, a new word will be received. 
If a unique combination is detected, the prefix will be checked, and stem and after that suffix will be checked 
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and stemmed, then again, a new word will be received. Figure 3 shows the activity diagram for 
the preprocessing process. 
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Figure 1. Use case diagram 


<category> 
<pattern> saya * </pattern> 
<template> 
<think> 
<set name="temp"><star/></set> 
</think> 
<condition name="temp"> 
<li value = "dosen"> 
<think><set name="topic">start</set><set name="tempS">job</set><set name="job">dosen</set></think> 
Sebagai dosen, apa yang anda butuhkan/harapkan dari «get name="apps"/>? |<get name="job"/> 
</li> 
<li value = "guru"> 
<think><set name="topic">start</set><set name="tempS">job</set><set name="job">dosen</set></think> 
Sebagai dosen, apa yang anda butuhkan/harapkan dari «get name="apps"/>? |<get name="job"/> 
</li> 
<li value = "pengajar"> 
<think><set name="topic">start</set><set name="tempS">job</set><set name="job">dosen</set></think> 
Sebagai dosen, apa yang anda butuhkan/harapkan dari «get name="apps"/>? |<get name="job"/> 
</li> 
<li value = "mahasiswa"> 
<think><set name="topic">start</set><set name="tempS">job</set><set name="job" >mahasiswa</set></think> 
Sebagai mahasiswa, apa yang anda butuhkan/harapkan dari «get name="apps"/>? |<get name="job"/> 


</li> 


<li value = "staff"> 
<think><set name="topic">start</set><set name="tempS">job</set><set name="job">staff</set></think> 
Sebagai staff, apa yang anda butuhkan/harapkan dari «get name="apps"/>? |<get name="job"/> 

</li> 

<li value = "baak"> 
<think><set name="topic">start</set><set name="tempS">job</set><set name="job">staff</set></think> 
Sebagai staff, apa yang anda butuhkan/harapkan dari «get name="apps"/>? |<get name="job"/> 

</li> 

<li value = "bauk"> 
<think><set name="topic">start</set><set name="tempS">job</set><set name="job">staff</set></think> 
Sebagai staff, apa yang anda butuhkan/harapkan dari <get name="apps"/>? |<get name="job"/> 

</li> 

<li> 
<random> 

<li>Maaf. Profesi anda apa? 
Dosen, mahasiswa, atau staff?</li> 

</random> 

</li> 

</condition> 
</template> 
</category> 





Figure 2. Example of AIML 
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Figure 3. Activity diagram preprocessing process 


2.4. User story 

The user story concept is a part of the agile development method that was first proposed in extreme 
programming for gathering user requirements that are needed by the system [4]. It then becomes key in 
the software development process [23]. User stories focus on the interaction between the user or 
customer [3], and they will describe functions that are needed for the system in the form of a story [4]. 
Example form of the user story can be seen in Figure 4 [24]: 
a. <role> represents the role of the person that provide the story. 
b. <activity> represents the requirement. 
c. <business value> represents the reason or value of the above requirement. 


As a <role> I can <activity> so that <business value» 





Figure 4. User story 


3. RESULTS AND ANALYSIS 

The result of the research 1s a chatbot application that used for gathering software requirements 
using user stories and Nazief & Adriani stemming algorithm. In this chatbot AIML used for the chatbot 
knowledge. A user acceptance test is applied in this research to evaluate the chatbot's performance, perceived 
by the users. 


3.1. Chatbot implementation 

Figure 5 (a) display the splash screen of the chatbot applications. Splash screen will be show when 
user open the applications. Figure 5 (b) shows the chat room list menu. This menu show list chat room that 
user created. In this menu user can create or delete chat room. Chat room is represented by a card. Each card 
of display index of card, the name of the application and role of user. 
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Figure 6 shows the user story menu. The user story menu is for the developer so they can see a list 
of user stories that have been obtained from the user. Each user story represented by a card that every card 
shows the story, reason, application name, and the date that the user story obtained. In each card also have 
a delete button so the developer can delete the user story. At the top of the user story, there is a field that can 
be used by the developer to search the user story by application name, story, and reason. This list of the user 
story is already sorted by descending so the latest data will always be displayed in the first row, Figure 7 
shows the chat rooms and an example of giving a user story. In this chat room, users can chat with the bot 
and give the user story. The first-time bot will ask the role of the user. After the user tells their bot, the role 
and the bot will then store the role information, and the user can start to give the user story. Every time 
the user gives the user the reason the bot will ask confirmation whether or not the user has given by the user 
correctly. If the story and the reason already correct, the user story will be accepted and saved. 


3.2. Chatbot metrics 

Several metrics were used to evaluate the bot’s performance quantitatively. The metrics used 
include total elapsed time, total number of user turn, total number system turn, the total number of turns per 
task, dan total elapsed time per turn. Total elapsed time calculates the total elapsed time from when the user 
starts the conversation until when the user finishes submit a user story. The total number of user turn 
calculates the number of user chats from when the user starts the conversation until when the user finishes 
submit a user story. The total number of system’s turn calculates the number of bot responses from when 
the user starts the conversation until when the user finishes submit a user story. Total elapsed time per turn 
calculates the elapsed time for a user to finish his/her turn; this is calculated by dividing the total elapsed time 
by the total number of turns. The metrics evaluate the provision of 30 user stories given by three users in 
which the time is measured manually using a timer. Table 1 shows the sample metric of a user when testing 
the chatbot application, performing a total of 8 tasks. Each task in Table 1 represents an activity in which 
a user is communicating using the chatbot to provide user stories, with the following numbers are calculated 
for the metrics. 
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Figure 5. (a) Splashscreen (b) Chat room list menu 
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Figure 7. Chat room 
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Table 1. Chatbot metrics 


Total elapsed time Total number of Total number of Total turns per Total elapsed time per 
(seconds) user turn system turn task turn (seconds) 
Task 1 50.70 6 6 12 4.23 
Task 2 37.18 7 7 14 2.66 
Task 3 36.14 6 6 12 3.01 
Task 4 32.00 6 6 12 2.67 
Task 5 39.24 6 6 12 3.27 
Task 6 44.57 6 6 12 3.71 
Task 7 44.44 7 7 14 3.17 
Task 8 48.68 6 6 12 4.06 
Total 332.95 50 50 100 26.77 
Average 41.62 6 6 12 3.35 


3.3. Preliminary user acceptance test 

A preliminary user acceptance test was conducted to evaluate the perceived usefulness (PU), user 
attitude towards (A), behavioral intention to use (BI), and perceived ease of use (PEOU), following 
the technology acceptance model proposed by Davis [25]. The questionnaire uses a five-point Likert scale 
used in order to determine the level of user acceptance, which is comprised of strongly agree (5), agree (4), 
neutral (3), disagree (2), and strongly disagree (1). Thirty-three respondents participated in this questionnaire. 
In order to test the capabilities of the chatbot, we use a university information system, as the case study for 
which the users will provide feedback. 

User acceptance test is done to 33 users using a web-based university information system, called 
MyUMN, as a test case for which the users will provide user stories on. The respondent is MyUMN’s users 
which are student, lecturer or university staffs. The respondent’s age is above 20 years old, both male and 
female respondents. Firstly, respondents were asked to download the APK file for the chatbot application. 
Then, users experimented with the app, based on the application description explaining about user story and 
what the chatbot aims to perform. After the users tried and experimented with giving feedbacks regarding 
the MyUMN web application and communicating with the chatbot, questionnaires were distributed, and 
interviews were conducted to get feedback from the users, mainly focusing on the four aspects of 
the technology acceptance model mentioned previously. The questionnaires’ answer sheet is formed based on 
the intervals using the Likert Scale, with interpretation, as shown in Table 2 below. 

Based on the questionnaires filled by the users after using the chatbot application, we calculated 
the average score from the questionnaire for each factor in the technology acceptance model. Results are as 
follow, 83.79% for user attitude towards the chatbot application, 85.45% for perceived usefulness, 
84.55% for perceived ease of use, and 83.03% for users’ behavioral intention to use. The graphical result of 
user acceptance test can be seen in Figure 8. Interiews also shows that the users’ as stakeholders of 
the MyUMN web application. The overall results seems not to differ from one aspect to another, however if 
we look closely, perceived usefulness achieved the highest score, this aspect measures the users’ acceptance 
regarding how well the chatbot is perceived to be useful to support the requirements gathering activity 
(from the developers point of view) and the feedback provision activity (from the users’ perspective) 
by providing a possibility of a natural communication to structure the feedbacks following the user 
story format, eliciting the feedback providers’ role, their needs, and the business value regarding 
the needs/requirements provided. 





User Acceptance Test 
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Figure 8. User acceptance result 
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The behavioral intention to use scored the lowest of the four aspects, however, looking at 
the interpretation score shown in Table 2, it still managed to be able to be interpreted as Strongly Agree. 
Interviews conducted after the questionnaires-filling activity shows the users’ motivation and intention to use 
the chatbot again in the future. 


Table 2. User acceptance test score interpretation 


Interpretation Percentage Score Value 
Strongly Disagree 0%-19.99% 1 
Disagree 20%-39.99% 2 
Neutral 40%-59.99% 3 
Agree 60%-79.99% 4 
Strongly Agree 80%-100% 5 


4. CONCLUSION 

The conducted research has developed a chatbot application based on the Android mobile platform, 
to support the requirements gathering activity by providing ways to bridge the communication gap between 
the developers of a software product and their stakeholders. The requirements that are gathered are then 
naturally enforced by the chatbot to follow the user story format containing the users’ role, 
requirements/needs, and its business value. The chatbot also implemented the Nazief & Adriani stemming 
algorithm for the specific need of stemming words written in Bahasa Indonesia. Furthermore, it utilizes 
the Artificial Intelligence Markup Language for the bot’s knowledge base in which the bot response is 
decided, given a user chat has been submitted. A preliminary user acceptance test based on the technology 
acceptance model shows promising results. The test focused on evaluating four aspects of user acceptance 
which are the user attitude toward the chatbot application, perceived usefulness, perceived ease of use, and 
the users’ behavioral intention to use. The chatbot received scores for the four aspects as follow, 83.79% for 
user attitude towards the chatbot application, 85.45% for perceived usefulness, 84.55% for perceived ease of 
use, and 83.03% for users’ behavioral intention to use. Future works include employing a more advanced 
natural language processing method for feedbacks written in Bahasa Indonesia, developing the web 
dashboard for the superuser (developers) to manage the AIML brain file and manage submitted feedback by 
filtering and classifying them into meaningful information for the software engineers. Furthermore, a more 
thorough user acceptance test could also be conducted to further evaluate the chatbot’s performance and its 
users’ perspective toward the chatbot. 
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